Open
Conversation
Collaborator
Author
|
sachilles
force-pushed
the
upgrade-to-18.9.0
branch
from
7da57a5 to
3568674
Compare
Collaborator
Author
|
Issue could be related to dependency to package |
Collaborator
Author
|
It seems that doesn't matter if the |
Contributor
|
Reproduced on my env. In addition, upgrade from previous version launches with no such error (tested with 18.8.4 -> 18.9.0 local build). Reports of similar errors have been traced to the database being accessed before it was fully initialized or lacks migration. As far as I checked, the database is mostly empty (none of the tables or anything else derived from gitalb were created) after the gitlab container exits. running 18.8.4. The result is 1050 rows. psql -> \dt * on postgresql for 18.8.4 restored from production envnewly created 18.9.0. The result is only 64 rows, all of which are standard data that comes with PostgreSQL. psql -> \dt * postgresql for newly created 18.9.0 |
Contributor
|
My upgrade from 18.8.4 to 18.9.0 in my dev environment did work. In a new test installation I'm running into the same issue. Another finding - maybe unrelated (from gitlab 18.9.0 release page): I'm rebuilding with ruby 3.3.10 right now. |
Contributor
|
Database issue still present with ruby 3.3.10. Ruby 3.3 changes: delete assets/build/patches/ruby/0001-avoid-seeding_until-ruby3.3.0.patch |
Contributor
|
Looking into stack trace, If the behavior of |
Contributor
|
Well, I checked gitaly log (/var/log/gitlab/supervisor/gitaly.log*) and found that gitaly.socket is on strange path. There might be an update for gitaly config file or something else.
|
Contributor
|
I see 2 listen statements: time="2026-02-24T11:28:06.050Z" level=info msg="listening at address" address=/home/git/gitlab/tmp/sockets/private/gitaly.socket pid=980 schema=unix |
Contributor
Yeah I noticed that after I reported. Still not sure which is active (or both?) |
Contributor
|
puma and sidekiq also show: PG::UndefinedTable: ERROR: relation "application_settings" does not exist 2026-02-24 12:42:25,949 WARN exited: sidekiq (exit status 1; not expected) After that both are running. Is gitlab_setup slower? |
Contributor
|
Anyway, I think we can temporally move things forward by disabling call of |
Contributor
I think it's not. gitlab DB just did not set up correctly and everything stuck. puma and sidekiq can't connect to gitlab (or gitaly, gitlab-workhorse or anything else). |
Contributor
|
18.9.0 launch successfully with patch : 
I'm not sure but we may need to revert the change provided by this patch if not # in assets/runtime/functions Line 2219, migrate_database()
if [[ -z ${COUNT} || ${COUNT} -eq 0 ]]; then
echo "Setting up GitLab for firstrun. Please be patient, this could take a while..."
exec_as_git force=yes bundle exec rake gitlab:setup \
${GITLAB_ROOT_PASSWORD:+GITLAB_ROOT_PASSWORD=$GITLAB_ROOT_PASSWORD} \
${GITLAB_ROOT_EMAIL:+GITLAB_ROOT_EMAIL=$GITLAB_ROOT_EMAIL} >/dev/null
+ else
+ echo "Reverting patch lib/gitlab/gitaly_client/circuit_breaker.rb"
+ git apply -R "${GITLAB_BUILD_DIR}/patches/gitlabhq/0005-Disable-enabled-check-on-circuit_breaker.call.patch"
fi |
Contributor
checking gitaly v18.9.0 but I could find out no critical change (like rename of Also I noticed that Even I commented out the line, gitaly client get response like below: and |
Contributor
|
Upstream issue (thanks @th-2021) : https://gitlab.com/gitlab-org/gitlab/-/issues/591292 |
Contributor
|
18.9.1 doesn't solve this. Error still happening. |
Contributor
|
@kkimurak do you know how gitaly and gitlab:setup interact? I commented out gitaly and get the same error. |
Contributor
|
@th-2021 Which file / line did you commented out? Rake task If the database is empty (newly created), If succeed, continues to: -> |
Contributor
|
I changed the autostart to false in the supervisor config.
|
Contributor
|
There is an older issue that looks similiar: |
Contributor
|
It doesn't matter whether gitaly is actually running or not. |
Contributor
|
After the crash of gitlab:setup no tables were created: |
Contributor
|
There are also no messages with redis after that: While on my production system it looks like: |
Contributor
|
Thank you for confirmation. |
Contributor
|
Well, I found another idea that can apply to So we may be able to start gitlab by dropoing database before rake task if [[ -z ${COUNT} || ${COUNT} -eq 0 ]]; then
echo "Setting up GitLab for firstrun. Please be patient, this could take a while..."
+ echo "Before gitlab:setup, we will drop database ${DB_NAME}."
+ echo "It will start in 10 seconds. If you do not agree with it, please stop the container."
+ sleep 10
+ echo "= DROP DATABASE ${DB_NAME}.."
+ # we cannot drop database withing opening it so connect to default database 'postgres'
+ PGPASSWORD="${DB_PASS}" psql -h ${DB_HOST} -p ${DB_PORT} -U ${DB_USER} -d postgres -Atw -c "DROP DATABASE ${DB_NAME}"
+ echo "= Done."
+ echo "= executing rake task gitlab:setup.."
exec_as_git force=yes bundle exec rake gitlab:setup \
${GITLAB_ROOT_PASSWORD:+GITLAB_ROOT_PASSWORD=$GITLAB_ROOT_PASSWORD} \
${GITLAB_ROOT_EMAIL:+GITLAB_ROOT_EMAIL=$GITLAB_ROOT_EMAIL} >/dev/null
+ echo "= Done."It also worked in my env. |
Contributor
|
kkimurak/sameersbn-gitlab:18.9.0-5ded24f1 seems to work, but I get a lot of: My setup is incomplete, so I can currently only see the log messages. |
Contributor
|
Do you have an image for the second idea? |
Contributor
|
More messages from first idea: |
Contributor
|
Upstream issue got: |
Contributor
|
Upstream bug is being worked on for 18.10 with a backport to 18.9. |
Contributor
|
Thank you for test, I have pushed image for second idea |
Contributor
|
Upstream fix MR with is goind on: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/225503 If the fix is correct, we can release 18.9.0 by adding https://gitlab.com/gitlab-org/gitlab/-/merge_requests/225503.patch to assets/build/patches/gitlabhq/. |
Contributor
|
I created a local image with the fix as 0005-circuitbreaker-fix.patch During database load there are a lot of puma/sidekiq crashes. Should we stop them before gitlab:setup? |
Contributor
|
I haven't been able to consistently experience the symptom of puma and sidekiq to crash. It sometimes happens and sometimes not. case ${1} in
app:start)
/usr/bin/supervisord -nc /etc/supervisor/supervisord.conf &
SUPERVISOR_PID=$!
+ while ! test -e "/var/run/supervisor.sock" 2>/dev/null; do
+ echo "waiting supervisor to start"
+ sleep 1
+ done
+ set +e
+ supervisorctl stop sidekiq gitlab:puma
+ set -e
migrate_databaseI did not stop gitaly because it seems to need to be running. There are other supervisor-managed processes (such as nginx, sshd, and gitlab-workhorse) that can be stopped without any problems, but I have not tried them. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GitLab 18.9 Release (for more details see https://about.gitlab.com/releases/2026/02/19/gitlab-18-9-released/)